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Figure  2:  How  satellites  predict  bathymetry  (Amos,  J.  (2014, 
October  2) 


3.  Single  Image  Super  Resolution  on 
Bathymetry  Data 
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of  images  so  that  it  can  take  only  the  low  resolution 
image  and  yield  a  prediction  of  a  high  resolution 
image  estimation.  This  is  done  by  treating  the  pixels 
of  the  low  resolution  image  as  points  of  data,  and 
mathematically  interpolating  those  points  to  create  an 
estimate  of  what  the  high  resolution  image  would  look 
like 

image  pairs.  For  example,  if  a  high  resolution  image 
consists  of  3600  pixels,  and  a  low  resolution  image 
consists  of  100  pixels  (a  difference  of  a  scaling  factor 
of  6),  you  can  take  the  low  resolution  image  and  up¬ 
sample  the  pixels  by  interpolating  them  as  data  points 
to  artificially  create  a  high  resolution  estimation  of 
3600  pixels 

by  using  the  comparison  of  pre-learned  pairs  of  low 
resolution  and  high  resolution  images.  An  example  of 
this  estimation  is  shown  in  Figure  3. 
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Figure  3:  Super-resolution  using  bicubic  interpolation  to  produce  an  HR 
estimate  using  LR  data  (Images  produced  using  Matlab). 

Before  we  could  go  about  improving  the 


fundamental  differences  between  that  nature  of  image 
data  and  bathymetry  data.  For  instance,  bathymetry 
data  deals  in  elevation,  whereas  image  data  deals  in 
variation  of  color/brightness  (RGB  value).  This  could 
introduce  some  problems  regarding  interpolation; 
specifically,  how  the  data  points  in  the  z-direction  (the 
height)  would  be  treated.  There  are  many  different 
variations  of  the  super-resolution  technique,  and  one 
way  in  which  they  vary  is  in  regards  to  the  super¬ 
resolution  method.  For  bathymetry  data,  we  needed  a 
super  resolution  technique  that  would  preserve  sharp 
changes  -  an  interpolation  method  that  would  smooth 
out  the  data  as  little  as  possible.  This  is  because  the 
actual  ocean  floor  is  not  very  smooth  there  are  sharp 
peaks  and  valleys,  and  in  order  to  accurately  represent 
the  ocean  floor  with  our  high  resolution  data 
estimations,  we  must  use  a  super-resolution  technique 
that  would  smooth  out  these  peaks  and  valleys  as  little 
as  possible.  Bicubic  interpolation  has  the  ability  to 
produce  high  resolution  estimations,  however  there  is 
considerable  a  smoothing  effect  as  a  result.  An 
example  of  this  smoothing  is  shown  in  Figure  4. 
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Figure  4:  Bicubic  Interpolation  shows  the  ability  to  predict  HR 
bathymetry  from  LR  data  (similar  to  the  capability  of  with  image  data), 
however  there  is  considerable  smoothing  out  of  the  prediction  (Plots 
produced  using  Matlab). 


uncertainties.  There  were  a  number  of  things  we  were 
unsure  of  most  of  which  stemmed  from  the 
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4.  Super  Resolution  Forests 

The  super-resolution  algorithm  that  was 
chosen  to  conduct  our  experiments  on  was  called 

-Resolution  F 

and  Bischof,  (2015)).  The  code  to  run  this  algorithm 
was  modified  so  that  it  would  operate  on  bathymetry 
data  instead  of  image  data. 

The  SRF  algorithm  works  as  follows  (see 
Figures  5  and  6):  As  input,  you  have  pairs  of  low 
resolution  and  high  resolution  data.  There  are  two 
main  processes  -  training  and  testing.  80%  of  the  input 
data  is  used  for  the  training  process,  and  the  remaining 
20%  is  reserved  for  the  testing  process.  The  size  of  the 
low  resolution  data  is  10x10  points,  and  the  size  of  the 
high  resolution  data  is  60x60  points.  The  low 
resolution  data  is  taken  and  undergoes  bicubic 
interpolation,  which  is  a  common  form  of 
interpolation  used  on  images.  In  doing  so,  it  is  up- 
sampled  and  has  the  same  dimensions  as  the  high 
resolution  data.  This  new  60x60  interpolated  data  is 
labeled  as  the  bicubic  estimates.  This  estimated  data  is 
then  compared  to  the  original  high  resolution  data  and 
the  SRF  training  process  begins,  resulting  in  a  super¬ 
resolution  forest  (SRF)  model  which  is  created  from 
the  residuals  of  the  original  high  resolution  data  and 
the  up-sampled  data. 

Once  the  model  is  created,  the  testing  process 
begins,  and  the  reserved  20%  of  data  pairs  is  used  for 
testing.  Each  of  the  low  resolution  data  blocks  are  up- 
sampled  using  bicubic  interpolation.  The  super¬ 
resolution  model  is  then  applied  to  these  bicubic 
estimates,  yielding  the  final  high  resolution  estimates. 
Each  of  the  data  blocks  are  tested  with  the  remaining 
20%  percent  of  data  pairs.  After  the  forest  is  applied, 
the  final  high  resolution  estimates  are  created  and  are 
trimmed  to  have  the  dimensions  of  48x48.  This 
process  of  cropping  the  edges  is  a  result  of  the  fact 
that  this  SRF  processing  technique  uses  convolutional 
filters  that  cause  distortions  around  the  borders  of  an 
image.  To  compensate,  the  borders  are  removed, 
leaving  us  with  48x48  dimensions  of  undistorted  data. 
The  original  high  resolution  data  input  is  then 
trimmed  to  also  have  dimensions  of  48x48  in  order  to 
match  the  high  resolution  estimates  dimensions  for 


comparison,  and  finally,  a  PSNR  is  computed  and 
plotted  for  all  data  points. 

SRF  SR  Method 
Training  Process 


Sensor  1  Data  Sensor  2  Data 


Figure  5:  Process  1/2  (Training)  of  the  original  SRF  Algorithm 


SRF  SR  Method 
Testing  Process 
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Figure  6:  Process  2/2  (Testing)  of  the  original  SRF  Algorithm 
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Variance  Comparison  of  LR  Data  and  HR  Data 


5.  Changing  the  LR  data  variance 

On  the  bathymetry  data,  the  SRF  process  gave 
us  a  higher  PSNR  than  bicubic  interpolation  (by 
about  1  dB). 


Patch  Number 

Figure  7:  PSNR  comparison  (with  mean  scores)  between  Bicubic 
Interpolation  and  SRF 


Figure  7  shows  the  comparison  between  the 
PSNR  scores  for  all  points  of  data  having  gone 
through  both  the  bicubic  interpolation  and  SRF 
algorithms.  According  to  our  results,  the  mean  PSNR 
score  from  bicubic  interpolation  (44.7)  was  surpassed 
by  the  mean  PSNR  score  from  the  SRF  algorithm 
(45.6).  Although  it  is  a  slight  improvement,  the  team 
believed  that  we  could  do  even  better  than  a  1  dB 
increase  for  the  SRF  algorithm.  Due  to  the  smoothing 
nature  of  bicubic  interpolation,  there  is  a  loss  in  detail 
of  high  frequency  original  bathymetry  data  while  low 
frequency  detail  is  preserved.  This  caused  us  to 
examine  the  variance  of  the  data,  and  to  do  a 
comparison  of  before  and  after  bicubic  interpolation. 
If  we  could  notice  a  relationship  between  the  low 
resolution  data  and  the  high  resolution  data,  then  it 
could  help  us  in  estimating  what  high  resolution 
bathymetry  data  would  look  like  given  only  low 
resolution  bathymetry  data. 
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Figure  8:  Variance  Comparison  between  LR  and  FIR  bathymetry  data 

Figure  8  shows  a  comparison  between  the 
variance  of  the  low  resolution  data  against  the 
variance  of  the  high  resolution  data.  The  blue  points 
represent  variance  values  for  each  point  of  data,  the 
red  line  represents  an  ordinary  least  squares  line  (a 
common  way  to  plot  a  line  of  best  fit),  and  the  green 
line  represents  a  median  fit  line  (a  line  constructed  by 
using  the  medians  of  each  point  between  the  two  sets 
of  data,  respectively).We  observed  that  there  was  a 
positive  correlation  between  the  variances  (with  a 
slope  almost  equal  to  1).  Having  a  correlation  between 
the  variances  could  enable  us  to  predict  estimations  of 
high  resolution  data,  given  that  we  know  both  the  low 
resolution  data  and  the  slope  of  what  the  correlation 
should  be. 
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Figure  9:  Variance  Comparison  between  LR  and  down-sampled  FIR 
(bicubic  interpolation)  bathymetry  data 
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We  then  took  a  look  at  the  relationship 
between  the  variance  of  the  low  resolution  data  and 
the  variance  of  the  estimated  low  resolution  data  using 
bicubic  interpolation  to  down-sample  the  known  high 
resolution  data  (As  shown  in  Figure  9). There  were 
two  main  observations.  Firstly,  there  was  a  drop  in 
magnitude  of  the  variance  of  the  down-sampled  data 
compared  to  the  original  high  resolution  data. 
Secondly,  there  was  a  slight  decrease  in  slope-value. 
Knowing  that  the  variance  of  the  down-sampled  data 
was  decreasing  and  that  there  was  a  shift  in  slope 
(closer  to  1),  if  we  could  move  the  data  points  away 
from  the  mean  a  certain  distance  5  proportional  to  how 
far  those  data  points  are  from  the  mean,  then  we  could 
derive  a  simple  expression  to  modify  the  variance  of 
our  estimated  high  resolution  data  to  closer  represent 
how  it  should  appear  in  reality. 

The  mathematics  behind  this  modified 
variance  calculation  is  as  follows: 


The  variance  of  the  low  resolution  data 
(VarLR)  is 

1  N 

VarLR  =  —  ^(x*  —  x )2 


By  modifying  each  point  x*  by  a  quantity 
s(x  —  x),  we  can  have  the  LR  data  variance  (VarLR) 
be  equal  to  the  HR  data  variance  (VarHR). 

1  N 

VarHR  =  —  4  s(x;  -  x)  -  x)2 

i 

1  N 

=  —  +  sxi  —SX  —  x)2 

i 

N 

=  — ^Oi(l  +  s)  -  *(1  4  s))2 
i 

=  ^  'Y.t1 + s)2  -x)2 


=  (1  +s)2Jj^_(Xi  -x)2 

By  substitution, 

VarHR  =(l4s)2  VarLR 

And  solving  for  s  yields. .. 


ivarHR 

s  = 

\ 

IvarHR 

j VarLR 

VarLR  ' 

By  implementing  an  edited  variance  within 
certain  locations  of  the  SRF  algorithm  code,  we  could 
run  experiments  to  see  if  implementing  an  edited 
variance  could  have  any  improvement  on  SRF’s 
overall  mean  PSNR.  Because  bicubic  interpolation  is 
a  step  that  occurs  within  the  SRF  algorithm,  the  drop 
in  magnitude  of  variance  after  bicubic  interpolation 
gave  us  reason  to  experiment  with  changing  the 
variance  before  and  after  bicubic  interpolation  occurs 
within  the  algorithm. 

6.  The  7  trials 

Within  the  SRF  algorithm  code,  there  were  3 
locations  at  which  the  variance  could  be  edited.  The 
first  location  was  before  the  bicubic  estimates  were 
created  (before  the  SRF  is  learned  and  applied),  the 
second  location  was  after  the  bicubic  estimates  were 
created  (also  before  the  SRF  is  learned  and  applied), 
and  the  third  location  was  after  the  high  resolution 
estimates  were  created  (after  the  SRF  is  learned  and 
applied).  Diagrams  showing  all  3  locations  within  the 
training  and  testing  processes  of  the  SRF  algorithm 
are  shown  in  Figures  11  and  12. 

Since  there  were  3  locations  where  the 
variance  could  be  edited,  7  trials  could  be  conducted 
in  total  (the  first  3  locations  independently,  first 
location  in  combination  with  the  second  location  and 
then  the  third  location,  and  finally  all  three  locations 
simultaneously).  A  table  showing  which  data  would 


5 


have  its  variance  changed  for  each  trial  is  displayed  in 
Figure  10. 

Trial  1 

1:  Apply  to  LR  [10x10]  Data 

-before  upsampling  and  before  application  of  SRF 

Trial  2 

2:  Apply  to  HR  bicubic  [interpolated)  [60x60]  Data 

-before  application  of  SRF 

Trial  3 

3:  Apply  to  HR  estimated  [48x48]  Data 

-after  application  of  SRF 

Trial  4 

4:  Apply  to  LR  and  to  HR  bicubic  (interpolated)  Data 

Trial  5 

5:  Apply  to  LR  and  to  HR  estimated  Data 

Trial  6 

6:  Apply  to  HR  bicubic  and  to  HR  estimated  Data 

Trial  7 

7:  Apply  to  LR,  HR  bicubic,  and  to  HR  estimated  Data 

Figure  10:  Table  showing  where  the  variance  was  changed  along  the 
SRF  algorithm  code  for  each  trial,  as  well  as  what  data  is  the  in  put  for 

each  trial. 
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Figure  12:  Diagram  showing  the  locations  for  the  possible  changes  of 
variance  in  the  SRF  testing  process  for  all  trials. 


1.  Results 
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Figure  11:  Diagram  showing  the  locations  for  the  possible  changes  of 
variance  in  the  SRF  training  process  for  all  trials. 


Out  of  all  of  the  7  trials,  the  trial  that  yielded  the 
greatest  overall  improvement  from  the  mean  PSNR 
score  of  the  original  SRF  process  was  the  PSNR  score 
of  trial  1  (as  shown  in  Figure  13). 


Trial  1  -  PSNR  Values 


PaEch  Number 


Figure  13:  PSNR  comparison  for  Trial  1  (changing  the  variance  of  the 
low  resolution  data  before  it  undergoes  bicubic  interpolation). 
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This  meant  that  implementing  the  edited  variance 
before  the  bicubic  estimates  were  created  caused  the 
mean  PSNR  to  increase  the  most,  and  all 
combinations  where  either  the  variance  was  changed 
after  the  bicubic  estimates  were  created  or  after  the 
high  resolution  estimates  were  created  caused  the 
change  in  PSNR  to  be  less  than  that  of  trial  1 . 

8.  Discussion 

As  mentioned,  for  any  case  where  the  variance 
was  changed  after  initial  bicubic  up-sampling,  the 
change  in  mean  PSNR  was  minimal.  It  is  likely  that 
the  reason  for  this  is  that  changing  the  variance  after 
both  the  training  and  testing  processes  occur  will 
introduce  noise  into  the  data,  which  will  in  turn  cause 
the  high  resolution  estimates  to  stray  further  away 
from  the  ground  truth.  Changing  the  variance  early  on 
in  the  SRF  algorithm  code  (before  bicubic 
interpolation  occurs)  would  mean  to  edit  the  raw  data 
as  opposed  to  noisy  data  created  later  on  along  the 
code.  Although  trials  4,  5,  and  7  also  had  the  variance 
edit  occur  before  the  initial  bicubic  interpolation, 
because  these  trials  had  the  variance  of  the  data 
changed  a  second  (or  third)  time  later  in  the  code  after 
training/testing,  it  is  suspected  that  the  additional  edits 
introduced  noise  into  the  data  and  thus  worsened  the 
improvement  on  the  mean  PSNR. 

9.  Conclusion  and  Future  Remarks 

These  results  show  that  it  is  possible  to 
improve  estimated  high  resolution  bathymetry  data 
with  the  use  of  a  single  image  super-resolution 
technique,  and  the  ability  to  further  improve  the 
estimations  by  altering  the  variance  of  the  low 
resolution  data  as  input  in  the  training  and  testing 
processes  of  the  SRF  algorithm.  A  next  step  from 
here  would  be  to  investigate  a  way  to  estimate  the 
desired  variance  of  the  data  blocks  without  using 
“unknown”  high  resolution  data.  Another  way  to 
further  improve  the  accuracy  of  high  resolution  data 
estimation  would  be  to  investigate  how  to  adjust  the 
variance  spatially,  instead  of  using  one  overall 
variance  of  the  patch. 
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